Training device, training method, and program

ABSTRACT

A training device, which is used to cause a user to be trained to operate an extra part based on a measurement result of user&#39;s brain activity, notifies the user of information indicating a degree of activation of a brain area corresponding to the extra part, as information indicating the state of the brain activity of the user who imagines trying to operate the extra part.

BACKGROUND

1. Technical Field

The present invention relates to training for manipulating an extra partby brain activity.

2. Related Art

A system is known which detects a user's intention from braininformation acquired by a brain activity measurement device and sends anintention identifier for identifying the intention to a robot so as tooperate the robot (JP-A-2010-198233).

In the related art, a robot remote from the user is to be operated. Onthe other hand, an extra part may be subjected to an operation. Theextra part is an actuator mounted on the user's body and acting as if itis a part of the user's body. When the extra part is operated accordingto a degree of activation of a brain area corresponding to the extrapart, if the user can freely control the degree of activation, theoperation will be more accurate. However, when the progress and effectof training is checked using the actual extra part, it takes time andeffort for the work.

SUMMARY

An advantage of some aspects of the invention is to check the progressand effect of training of operating an extra part by user's brainactivity, without using an actual extra part.

The advantage can be achieved by the following configurations.

An aspect of the invention provides a training device used to cause auser to be trained to operate an extra part based on a measurementresult of user's brain activity. The training device includes anotification unit that notifies the user of information indicating adegree of activation of a brain area corresponding to the extra part, asinformation indicating the state of the brain of the user who imaginestrying to operate the extra part. According to this aspect, it ispossible to determine the progress and effect of training, without usingan actual extra part.

The training device may further include a display unit that displaysinformation indicating the degree of activation, as the notificationunit. With this configuration, the user can view the degree ofactivation.

The training device may be a head mounted display, and the display unitmay allow the user to view a virtual image. With this configuration, itis possible to perform training using display of a virtual image by thehead mounted display device.

In the training device, the display unit may allow the user to view animage on which the extra part is operating. With this configuration, itis possible to train to manipulate an extra part by user's brainactivity. Specifically, the user can perform the training by consideringthat the action is realized by the user's imagination while viewing theimage on which the extra part is operating.

In the training device, the display unit may display, as the image, afirst image that is an image from a first person viewpoint and a secondimage that is an image from a third person viewpoint. With thisconfiguration, training can be effectively performed by using images ofdifferent viewpoints.

The training device may be a transparent head mounted display; and thefirst image may be an image in which the extra part appears to bedisposed relative to the user's actual body. With this configuration,the user obtains information indicating the degree of activation, andfeels realistic when viewing the operation of the extra part. Inparticular, when viewing an image in which the extra part is operating,the user's brain is likely to recognize the extra part as a part of thebody, and the above training can be effectively performed.

The training device may cause the user to view an image in which theextra part appears to apply stimulation, when the stimulation is appliedto at least one of the tactile organ and the auditory organ of the user.With this configuration, illusion feeling can be given to the user'sbrain, and in turn, training can be performed effectively, by giving theuser stronger augmented reality. The illusion used herein is to feel asif there is an actual extra part.

The invention can be realized in various forms other than the above. Forexample, the invention can be realized in the form of a training method,a program for realizing the method, a non-temporary storage mediumstoring the program, or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an external view of an HMD.

FIG. 2 is a block diagram functionally illustrating the configuration ofthe HMD.

FIG. 3 is a diagram illustrating an aspect in which image light isemitted by an image light generation unit.

FIG. 4 is an external view of an electroencephalogram monitor.

FIG. 5 is a flowchart illustrating a corresponding area specifyingprocess.

FIG. 6 is a diagram illustrating an extra part.

FIG. 7 is a diagram illustrating an imaginary repetitive motion of theextra part.

FIG. 8 is a diagram illustrating another imaginary repetitive motion ofthe extra part.

FIG. 9 is a diagram schematically illustrating an activation area of abrain before training.

FIG. 10 is a flowchart illustrating a first training process.

FIG. 11 is a diagram illustrating a first image.

FIG. 12 is a view illustrating an image from a rear viewpoint.

FIG. 13 is a view illustrating an image from a front viewpoint in anupper diagonal direction.

FIG. 14 is a diagram illustrating an illusion image.

FIG. 15 is a diagram schematically illustrating an activation area aftertraining.

FIG. 16 is a flowchart illustrating a second training process.

FIG. 17 is a diagram illustrating a training image in the secondtraining process.

FIG. 18 is a flowchart illustrating the first training process(modification example).

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 illustrates an external view of an HMD 100. The HMD 100 is a headmounted display (HMD). The HMD 100 is an optically transparent typedevice that allows a user to view a virtual image and to simultaneouslydirectly view an outside view. The HMD 100 functions as a trainingdevice for the first and second training processes (described later). Inthis specification, a virtual image viewed by the user by the HMD 100 isalso referred to as a “display image” for convenience. The display imageis a moving image in the present embodiment.

The HMD 100 includes an image display unit 20 for allowing a user toview a virtual image in a state of being mounted on the head of a user,and a controller 10 for controlling the image display unit 20.

The image display unit 20 is a wearing object to be worn on the head ofthe user, and has a spectacle shape in the present embodiment. The imagedisplay unit 20 includes a right holding unit 21, a right displaydriving unit 22, a left holding unit 23, a left display driving unit 24,a right optical image display unit 26, a left optical image display unit28, a camera 61, and a microphone 63. The right optical image displayunit 26 and the left optical image display unit 28 are respectivelydisposed immediately before the user's right and left eyes when the userwears the image display unit 20. The one end of the right optical imagedisplay unit 26 and the one end of the left optical image display unit28 are connected to each other at a position corresponding to middle ofthe forehead of the user when the user wears the image display unit 20.

The right holding unit 21 is a member extending from the end ER which isthe other end of the right optical image display unit 26 to a positioncorresponding to the temporal part of the user when the user wears theimage display unit 20. Likewise, the left holding unit 23 is a memberextending from the end EL which is the other end of the left opticalimage display unit 28 to a position corresponding to the temporal partof the user when the user wears the image display unit 20. The rightholding unit 21 and the left holding unit 23 hold the image display unit20 on the head of the user like a temple of glasses.

The right display driving unit 22 and the left display driving unit 24are disposed on the side facing the head of the user when the user wearsthe image display unit 20. In the following description, the rightdisplay driving unit 22 and the left display driving unit 24 will becollectively and simply referred to as the “display driving unit”, andthe right optical image display unit 26 and the left optical imagedisplay unit 28 will be collectively and simply referred to as “opticalimage display unit”.

The display driving units 22 and 24 include liquid crystal displays 241and 242 (hereinafter, also referred to as “LCDs 241 and 242”),projection optical systems 251 and 252, and the like (see FIG. 2).Details of the configuration of the display driving units 22 and 24 willbe described later. The optical image display units 26 and 28 as opticalmembers include light guide plates 261 and 262 (see FIG. 2) and a lightcontrol plate. The light guide plates 261 and 262 are formed of a lighttransparent resin material or the like, and guide the image light outputfrom the display driving units 22 and 24 to the user's eyes. The lightcontrol plate is a thin plate-like optical element, and is disposed soas to cover the front side of the image display unit 20, which is theside opposite to the user's eye side. The light control plate protectsthe light guide plates 261 and 262, and suppresses damage to the lightguide plates 261 and 262, attachment of dirt, and the like. Further, itis possible to adjust the easiness of visibility of virtual images byadjusting the amount of external light entering the eye of the user, byadjusting the light transmittance of the light control plate. The lightcontrol plate can be omitted.

The camera 61 is disposed at a position corresponding to the middle ofthe forehead of the user when the user wears the image display unit 20.Therefore, in a state in which the user wears the image display unit 20on the head, the camera 61 captures the outside view and acquires thecaptured image. The outside view is the scenery outside the user's gazedirection. The camera 61 is a monocular camera. The camera 61 may be astereo camera.

The microphone 63 acquires sound. The microphone 63 is disposed on theopposite side (outer side) of the side facing the user of the rightdisplay driving unit 22 when the user wears the image display unit 20.

The image display unit 20 further includes a connecting portion 40connecting the image display unit 20 to the controller 10. Theconnecting portion 40 includes a main body cord 48, a right cord 42, aleft cord 44, and a connecting member 46 which are connected to thecontroller 10. The right cord 42 and the left cord 44 are cords in whichthe body cord 48 is branched into two cords. The right cord 42 isinserted into the casing of the right holding unit 21 from the front endportion AP in the extending direction of the right holding unit 21 andis connected to the right display driving unit 22. Similarly, the leftcord 44 is inserted into the casing of the left holding unit 23 from thefront end portion AP in the extending direction of the left holding unit23, and is connected to the left display driving unit 24. The connectingmember 46 is provided at a branch point among the main body cord 48, theright cord 42, and the left cord 44, and has a jack for connecting theearphone plug 30. The right earphone 32 and the left earphone 34 extendfrom the earphone plug 30.

The image display unit 20 and the controller 10 transmit various signalsthrough the connecting portion 40. Connectors (not illustrated) to beengaged with each other are respectively provided in the end portion onthe opposite side of the connecting member 46 of the body cord 48 andthe controller 10. The controller 10 and the image display unit areconnected or disconnected by engaging/disengaging between the connectorof the body cord 48 and the connector of the controller 10. For example,a metal cable or an optical fiber may be adopted for the right cord 42,the left cord 44, and the main body cord 48.

The controller 10 controls the HMD 100. The controller 10 includes adetermination key 11, a lighting unit 12, a display switching key 13, atrack pad 14, a brightness switching key 15, a direction key 16, a menukey 17, and a power switch 18. When a press operation of thedetermination key 11 is detected, a signal for determining the contentoperated by the controller 10 is output. The lighting unit 12 notifiesof the operating state of the HMD 100 as its light emission state.Examples of the operating state of the HMD 100 include power supplyON/OFF. For example, an LED is used as the lighting unit 12. When apress operation of the display switching key 13 is detected, forexample, a signal for switching the display mode of the content movingimage to 3D or 2D is output. When the operation of the user's finger onthe operation surface of the track pad 14 is detected, a signalcorresponding to the detection content is output. Various track padssuch as an electrostatic type, a pressure detection type, and an opticaltype may be adopted as the track pad 14. When a pressing operation ofthe brightness switching key 15 is detected, a signal for increasing ordecreasing the brightness of the image display unit 20 is output. When apressing operation corresponding to each of up, down, right, and leftdirections of the direction key 16 is detected, a signal correspondingto the detected contents is output. When the slide operation of thepower switch 18 is detected, the power-on state of the HMD 100 isswitched.

FIG. 2 is a block diagram functionally illustrating the configuration ofthe HMD 100. As illustrated in FIG. 2, the controller 10 includes astorage 120, a power supply 130, an operation unit 135, a wirelesscommunication unit 132, a scenario database 138 (scenario DB 138), a CPU140, an interface 180, a transmission unit 51 (Tx 51) and thetransmission unit 52 (Tx 52). The operation unit 135 receives anoperation by the user. The operation unit 135 includes a determinationkey 11, a display switching key 13, a track pad 14, a brightnessswitching key 15, a direction key 16, a menu key 17, and a power switch18.

The power supply 130 supplies power to each part of the HMD 100. Forexample, a secondary battery may be used as the power supply 130. Thewireless communication unit 132 executes wireless communication withother devices such as, for example, a content server, a television, anda personal computer (PC), conforming to a predetermined wirelesscommunication standard such as a wireless LAN or Bluetooth (registeredtrademark).

The storage 120 includes a ROM, a RAM, and the like. Various programsare stored in the ROM of the storage 120. The CPU 140, which will bedescribed later, reads various programs from the ROM of the storage 120and stores them in the RAM of the storage 120, thereby executing variousprograms.

The scenario DB 138 stores an AR scenario which is a plurality of movingimages including an augmented reality (AR) image. The AR scenario in thepresent embodiment is a moving image including an AR image displayed onthe image display unit 20 and a sound output through the earphones 32and 34. Further, the HMD 100 can execute not only the AR scenario storedin the scenario DB 138, but also the AR scenario received from anotherdevice through the wireless communication unit 132.

The CPU 140 can control the operating system 150 (OS 150), the displaycontroller 190, the sound processor 170, the image processor 160, andthe display image setting unit 165 by reading out and executing theprogram stored in the ROM of the storage 120.

The display controller 190 generates control signals for controlling theright display driving unit 22 and the left display driving unit 24.Specifically, the display controller 190 controls the driving of theright LCD 241 by the right LCD controller 211, the driving of the rightbacklight 221 by the right backlight controller 201, the driving of theleft LCD 242 by the left LCD controller 212, the driving of the leftbacklight 222 by the left backlight controller 202, or the like, inresponse to the control signal, respectively. Thus, the displaycontroller 190 controls generation and emission of image light by theright display driving unit 22 and the left display driving unit 24,respectively. For example, the display controller 190 causes both theright display driving unit 22 and the left display driving unit 24 togenerate image light, only one of them to generate image light, or bothof them not to generate image light. Generating image light is alsoreferred to as “displaying an image”.

The display controller 190 transmits the control signals for the rightLCD controller 211 and the left LCD controller 212 through thetransmission units 51 and 52, respectively. In addition, the displaycontroller 190 transmits control signals to the right backlightcontroller 201 and the left backlight controller 202, respectively.

The image processor 160 acquires an image signal included in thecontent, and transmits the acquired image signal to the reception units53 and 54 of the image display unit 20 through the transmission units 51and 52. The image processor 160 may perform image processes such as aresolution conversion process, various color tone correction processessuch as brightness and saturation adjustment, a keystone correctionprocess, and the like on the image data as necessary.

The sound processor 170 acquires a sound signal included in the content,amplifies the acquired sound signal, and supplies the amplified soundsignal to the speaker (not illustrated) in the right earphone 32 and thespeaker (not illustrated) in the left earphone 34, connected to theconnecting member 46. For example, when adopting the Dolby (registeredtrademark) system, the processes are performed on the sound signal, andfor example, different sounds, whose frequencies and the like arechanged, are output from the right earphone 32 and the left earphone 34,respectively. In addition, the sound processor 170 transmits the soundacquired by the microphone 63 to the display image setting unit 165 as acontrol signal.

The sound processor 170 outputs sounds based on the sound signalsincluded in another AR scenario, through the earphones 32 and 34. Thesound processor 170 outputs sounds based on the sound signals includedin the AR scenario, through the earphones 32 and 34. The sound processor170 executes various processes, based on the sound acquired from themicrophone 63. For example, when options are included in the ARscenario, the sound processor 170 selects one option from the optionsbased on the acquired sound.

The interface 180 is used to connect various external apparatuses OAwhich are content supply sources to the controller 10. Examples of theexternal apparatuses OA include a storage device that stores an ARscenario, a PC, a mobile phone, or the like. For example, a USBinterface, a micro USB interface, a memory card interface, or the likemay be used as the interface 180.

The image display unit 20 includes a right display driving unit 22, aleft display driving unit 24, a right light guide plate 261 which is aright optical image display unit 26, a left light guide plate 262 whichis a left optical image display unit 28, a camera 61, and a microphone63.

The right display driving unit 22 includes a reception unit 53 (Rx 53),a right backlight controller 201 (right BL controller 201) and a rightbacklight 221 (right BL 221) which function as light sources, a rightLCD controller 211 and a right LCD 241 which function as displayelements, and a right projection optical system 251. The right backlightcontroller 201 and the right backlight 221 function as light sources.The right LCD controller 211 and the right LCD 241 function as displayelements. The right backlight controller 201, the right LCD controller211, the right backlight 221, and the right LCD 241 are collectivelyreferred to as “image light generation unit”.

The reception unit 53 functions as a receiver for serial transmissionbetween the controller 10 and the image display unit 20. The rightbacklight controller 201 drives the right backlight 221 based on theinput control signal. The right backlight 221 is, for example, a lightemitter such as an LED or an electroluminescence (EL). The right LCDcontroller 211 drives the right LCD 241, based on the control signalstransmitted from the image processor 160 and the display controller 190.The right LCD 241 is a transparent liquid crystal panel in which aplurality of pixels are arranged in a matrix.

The right projection optical system 251 is formed of a collimator lensthat converts the image light emitted from the right LCD 241 into aparallel light flux. The right light guide plate 261 as the rightoptical image display unit 26 guides the image light output from theright projection optical system 251 to the user's right eye RE, whilereflecting it along a predetermined optical path.

The left display driving unit 24 has the same configuration as that ofthe right display driving unit 22. The left display driving unit 24includes a reception unit 54 (Rx 54), a left backlight controller 202(left BL controller 202) and a left backlight 222 (left BL 222) whichfunction as light sources, a left LCD controller 212 and a left LCD 242which function as the display elements, and a left projection opticalsystem 252. The left backlight controller 202 and the left backlight 222function as light sources. The left LCD controller 212 and the left LCD242 function as display elements. The left backlight controller 202, theleft LCD controller 212, the left backlight 222, and the left LCD 242are collectively referred to as “image light generation unit”. Further,the left projection optical system 252 is formed of a collimator lenswhich converts the image light emitted from the left LCD 242 into aparallel light flux. The left light guide plate 262 as the left opticalimage display unit 28 guides the image light output from the leftprojection optical system 252 to the user's left eye LE, whilereflecting it along a predetermined optical path.

FIG. 3 illustrates an aspect in which image light is emitted by an imagelight generation unit. The right LCD 241 converts the illumination lightIL emitted from the right backlight 221 into effective image light PLrepresenting an image, by changing the transmittance of the lighttransmitted through the right LCD 241 by driving the liquid crystals atrespective pixel positions arranged in a matrix. The same is applied tothe left side. As described above, in the present embodiment, abacklight system is adopted. However, the image light may be emittedusing the front light system or a reflection system.

FIG. 4 illustrates an external view of an electroencephalogram monitor300. The electroencephalogram monitor 300 is used to acquire anelectroencephalogram (EEG), and includes a sensor unit 310 and adetection circuit 320.

The sensor unit 310 has a shape like a hat and is attached to the user'shead. In the sensor unit 310, a plurality of electrodes are disposed onthe mounting surface. A variation in potential corresponding to thebrain area having each electrode disposed is acquired from eachelectrode. The detection circuit 320 acquires and stores anelectroencephalogram in each part where the electrode is disposed, byprocessing the acquired variation of the potential. The detectioncircuit 320 includes an interface (not illustrated) for outputting thestored electroencephalogram to an external device.

FIG. 5 is a flowchart illustrating a procedure of a corresponding areaspecifying process. First, as a preliminary measurement procedure, anelectroencephalogram of an examinee who is the user when the examineeimagines operating an extra part E (FIG. 6) with his/her own will isacquired (S400).

In S400, measurement by the electroencephalogram monitor 300 is used. InS400, display by the HMD 100 is not used. In other words, the HMD 100may not be worn by the examinee, or nothing may be displayed in a statewhere the HMD 100 is worn by the examinee. S400 is executed to acquirean electroencephalogram before training.

FIG. 6 illustrates an extra part E. In the present embodiment, theactual extra part E is not used. That is, the extra part E illustratedin FIG. 6 indicates an image imaged by the examinee or an image viewedby the examinee through the HMD 100. In this embodiment, the extra partE is a third arm. The third arm as used here is derived from the back ofthe right shoulder, and is an imaginary body part including the upperarm, the forearm, and the hand. The hand as used here is a part from thewrist to the fingertip, and includes five fingers, a palm of the hand,and the back of the hand.

In S400, the examinees were instructed to imagine that extra parts E aregenerated in their bodies and operating the extra part E with their ownwill, from those who assist training (hereinafter referred to asassistants).

FIG. 7 and FIG. 8 illustrate imaginary repetitive motions of the extrapart E. As illustrated in FIG. 7 and FIG. 8, in S400, the examineeimagines moving the extra part E front and rear.

FIG. 9 schematically illustrates activation areas of the brains beforetraining, with respect to three examinees A, B, and C. The activationarea is an active brain area. Specifically, the activation area is abrain area in which an increase in blood flow occurs, or brain cellignition (spike) occurs actively.

As illustrated in FIG. 9, before the training, the activation areas ofthe examinees A, B, and C are different from each other. Furthermore, inthe case of examinees A, B, C-person-person, the activation area iswide.

As described above, the occurrence of individual differences in theactivation areas or the expansion of the activation areas over a widerange is a phenomenon that may be observed when an unaccustomedoperation is performed.

Subsequently, the first training process (S500) is executed. FIG. 10 isa flowchart illustrating the procedure of the first training process.The operation of the HMD 100 in the first training process is realizedby the CPU 140 executing the program stored in the storage 120. The HMD100 displays the display image in the first training process, by playingthe scenario stored in the scenario database 138. The HMD 100 starts thefirst training process, with the instruction input through the operationunit 135 as a trigger. The examinee performs an input operation on theoperation unit 135 in response to the instruction of the assistant.

First, a viewpoint is selected (S510). S510 is realized by the examineeoperating the operation unit 135. The examinee performs an inputoperation on the operation unit 135 in response to the instruction ofthe assistant. In the present embodiment, a first person viewpoint and aplurality of third person viewpoints are prepared.

FIG. 11 illustrates a display image (a first image) when the firstperson viewpoint is selected. The first person viewpoint is the actualviewpoint of the examinee. A rectangle drawn with a solid line indicatesa boundary between the optical image display units (the right opticalimage display unit 26 and the left optical image display unit 28) viewedby the examinee.

The examinee views the extra part E as the display image (first image).In other words, the examinee experiences the augmented reality by thefirst image.

Both hands drawn by solid lines are the examinee's own hands seenthrough the optical image display unit. The forearm drawn by a brokenline is the examinee's own forearm which is viewed without passingthrough the optical image display unit. His/her own body which is seenthrough the optical image display unit is not a display image.

On the other hand, the third person viewpoint is used to view examinee'sown body from a third person, as exemplified in FIGS. 6, 7, and 8. Inother words, the display image (second image) based on the third personviewpoint gives an examinee a feeling of viewing his/her own body as ifhe/she withdraws from the body. Therefore, it is possible for theexaminee to view the image of the back, which is difficult to be vieweddirectly with the naked eye. The viewpoint illustrated in FIG. 6 isselectable in S510, and is a rear viewpoint in an upward diagonaldirection. The viewpoints illustrated in FIG. 7 and FIG. 8 areselectable in S510, and are viewpoints in an upward direction. FIG. 12illustrates a rear viewpoint as another viewpoint selectable in S510.FIG. 13 illustrates a front viewpoint in an upper diagonal direction asanother viewpoint selectable in S510. In S510, any one of theseviewpoints is selected. In S510, the display image described so far isnot yet displayed.

If the viewpoint is selected, display of an illusion image and tactilestimulation are performed (S520). FIG. 14 illustrates an illusion image.Regardless of the selection in S510, an image illustrated in FIG. 14 isdisplayed as a display image from the rear viewpoint, as the illusionimage. The stimulation device 900 illustrated in FIG. 14 is notdisplayed on the illusion image, but is illustrated in FIG. 14 for thepurpose of explanation. The stimulation device 900 is attached to theexaminee's back in advance.

The illusion image is a moving image in which the extra part E appearsto come into contact with or release from the back as the tactile organof the examinee. Then, at a timing when the extra part E appears to comeinto contact with the back of the examinee, tactile stimulation isapplied to the back of the examinee by the stimulation device 900. Thestimulation device 900 applies tactile stimulation to the examinee, bygenerating vibration. The stimulation device 900 is wirelessly connectedto the controller 10, and acquires from the controller 10, informationindicating the timing at which the extra part E appears to be touchingthe examinee's back, thereby realizing the vibration at the abovetiming.

S520 is a step for causing the examinee to feel illusion. The illusionused herein is that the examinee feels the extra part E as a part ofhis/her own body. Training in a state that the examinee feels illusionas described above (hereinafter, illusion state) increases the effect oftraining. S520 is completed after a predetermined time from the start.

Subsequently, display of the training image is started (S530). Thetraining image is an image from the viewpoint selected in S510, and is amoving image in which the extra part E is operating. The operation ofthe extra part E is, for example, the operation illustrated in FIGS. 7and 8 in the case of a viewpoint in an upward direction. In S530, theexaminee is instructed by the assistant to imagine that the extra part Eto be viewed is to operate on his/her own will.

Next, the assistant determines whether or not the examinee is in anillusion state (S540). The assistant executes S540 during display of thetraining image. The assistant observes the electroencephalogram acquiredby the electroencephalogram monitor 300 and determines whether or notthe examinee is in an illusion state. Waveforms peculiar toelectroencephalograms are known to appear in the illusion state. Theassistant performs S540 based on the presence or absence of thiswaveform.

The assistant displays information indicating the activation areaacquired by the electroencephalogram monitor 300, on the monitor of thePC as an image, for the purpose of S540. In the image to be displayed,the activation area is highlighted by color coding or the like. For thisimage, a standard brain is used. The standard brain is an image of abrain prepared in advance as an average brain. The PC executes a processof fitting the information acquired by the electroencephalogram monitor300 to the case of the standard brain.

If the assistant determines that it is not in an illusion state (NO inS540), the assistant instructs the examinee to return to S520 and repeatto display the illusion image and apply the illusionary stimulation, byoperating the operation unit 135.

On the other hand, if it is determined that it is in an illusion state(YES in S540), the assistant does not give an instruction to theexaminee, continues the display of the training image to continuetraining.

If the examinee instructs change of the viewpoint by operating theoperation unit 135 according to the instruction of the assistant, whilethe display of the training image is continued (change of viewpoint inS550), the process returns to S510 and the viewpoint is selected again.The assistant observes the electroencephalogram during training andinstructs the viewpoint to be changed as appropriate so that trainingcan be performed more effectively.

If the examinee instructs the completion of the training by operatingthe operation unit 135 according to the instruction of the assistant,while the display of the training image is continued (completion inS550), the first training process is completed.

If the first training process is ended, as a measurement procedure, theelectroencephalogram of the examinee when the examinee imaginesoperating the extra part E with his/her own will is acquired (S600).S600 is a step having the same contents as S400.

Subsequently, as an effect measurement procedure, the assistantdetermines whether or not the activation area is converged (S700). Theassistant sees the image representing the activation area, anddetermines whether or not the activation area is converged, by using thefollowing standard. The standard includes whether or not the activationarea of the examinee to be determined is roughly the same as theactivation areas of other examinees of a predetermined number of peopleor more, and whether or not the activation area is narrower than beforethe training.

FIG. 15 schematically illustrates activation areas which are converged,with respect to three examinees A, B, and C. In the case illustrated inFIG. 15, the activation areas of the brains of the examinees A, B, and Care roughly the same, and the activation areas are narrower than beforethe training. In such a state, the assistant determines that theactivation area is converged for each of the examinees A, B, and C.

As described above, it is thought that the reason why the activationareas are approximately the same after training is that there is hardlyindividual difference in the brain area where the brain area to beactivated to operate a certain body part in the primary motor cortex.From the results illustrated in FIG. 15, it can be said that there ishardly any individual difference even if it is a third arm that does notactually exist.

In addition, it is considered that the reason why the activation areabecomes narrower than before training is getting accustomed to imaginingthe operation by training. It is said that the degree of activation ofthe entire brain decreases as an examinee gets used to an operation, intraining of actually operating the examinee's body, as a differenttraining from the present embodiment. In other words, if the examineegets accustomed to the operation, the operation is realized by localactivation in the primary motor cortex, and the burden on the brain canbe reduced. From the results illustrated in FIG. 15, it can be said thatthe same phenomenon can be seen even in the case of the extra part E.

If the assistant determines that the activation area is not converged(NO in S700), the process returns to S500, and the first trainingprocess (S500) is executed again. On the other hand, if the assistantdetermines that the activation area is converged (YES in S700), theeffect of the first training process is checked in the effectmeasurement procedure (S700).

Thereafter, as a specific procedure, a brain area corresponding to theextra part E which is the third arm is specified from the measurementresult illustrated in FIG. 15 (S800). Specifically, the brain areaactivated in the measurement procedure (S600) after the training isspecified as the brain area corresponding to the extra part E.Hereinafter, the brain area specified in this manner is referred to as acorresponding area.

FIG. 16 is a flowchart illustrating the procedure of the second trainingprocess. The second training process is executed after the activationarea is specified by the corresponding area specifying process. Thesecond training process is similar to the first training process, andthe steps having the last two digits of the step number havesubstantially the same contents. In the following description, contentsnot specifically described are the same as in the first trainingprocess.

The operation of the HMD 100 in the second training process is realizedby the CPU 140 executing the program stored in the storage 120. The HMD100 displays the display image in the second training process, byplaying the scenario stored in the scenario database 138. The HMD 100starts the second training process, with the instruction input throughthe operation unit 135 as a trigger. Since the examinee is accustomed tothe handling of the HMD 100 through the first training process, he/shevoluntarily executes an input operation for the operation unit 135.

First, a viewpoint is selected (S610). The selection of the viewpoint isthe same as in the first training process. When the viewpoint isselected, display of an illusion image and tactile stimulation areexecuted (S620). The display of the illusion image and the tactilestimulation are also the same as in the first training process.

Subsequently, display of the training image (see FIG. 17) is started(S630). An indication of a degree of activation is included in thetraining image in the second training process, in addition to the movingimage of the extra part E in the first training process. Here, thedegree of activation used herein is the degree of activation of thecorresponding area. The degree of activation is an index indicating thestate of the brain activity of the user who imagines trying to operatethe extra part E.

FIG. 17 exemplifies a training image in the second training process.FIG. 17 illustrates a training image when the first person viewpoint isselected. This training image includes a gauge 700 as an indication ofthe above-mentioned degree of activation. The gauge 700 becomes longeras the degree of activation increases, and becomes shorter when thedegree of activation decreases.

In the configuration in which the actual extra part E is operated basedon the brain activity of the corresponding area, it is preferable thatthere is a difference in the brain activity of the corresponding area asmuch as possible between the case where the user imagines trying tooperate the extra part E and the case where the user does not imaginetrying to operate the extra part E.

Therefore, training is performed in order to generate a difference inthe brain activity of the corresponding area as much as possible betweenthe case where the user imagines trying to operate the extra part E andthe case where the user does not imagine trying to operate the extrapart E, by allowing the examinee to view the degree of activation of thecorresponding area.

The controller 10 acquires information indicating the degree ofactivation of the corresponding area from the PC through wirelesscommunication. The PC extracts information indicating the degree ofactivation of the corresponding area, from the information acquired fromthe detection circuit 320 of the electroencephalogram monitor 300, andinputs it to the controller 10.

Next, the examinee himself determines whether or not the examinee is inan illusion state (S640). The determination method is the same as S540.

If it is determined that it is not in an illusion state (NO in S640),the examinee operates the operation unit 135, thereby returning to S620to repeat the display of the illusion image and the application of theillusion stimulation. On the other hand, if it is determined that it isin the illusion state (YES in S640), the display of the training imageis continued and the training is continued.

If the change of the viewpoint is instructed by the examinee operatingthe operation unit 135 while the display of the training image iscontinued (change of viewpoint in S650), the process returns to S610 andthe viewpoint is selected again. The examinee observes theelectroencephalogram during training on the monitor of the PC, andchanges the viewpoint as appropriate so that training can be performedmore effectively.

If the completion of training is instructed by the examinee operatingthe operation unit 135 while the display of the training image iscontinued (completion in S650), the second training process iscompleted.

According to the above-described embodiment, at least the followingeffects can be achieved.

The corresponding area can be specified by the first training process.Since the corresponding area can be specified, it is possible toconstruct a system that allows the user to operate the actual extra partE based on the degree of activation of the corresponding area.

Furthermore, the examinees who have undergone the first training processcan operate the actual extra parts E with their electroencephalogram.Since such a system controls the operation of the extra part by thelocal electroencephalogram in the primary motor cortex, differently fromthe configuration of detecting the intention of the user based on theactivity of the entire brain in the related art, thereby controlling theoperation more accurately than in the past.

It is possible to perform training of increasing the degree ofactivation of the corresponding area, by the second training process. Inturn, the operation of the actual extra part E can be more preciselycontrolled.

The training contents of the first and second training processes can beexecuted without using the actual extra part E. Therefore, training canbe easily performed.

Since the HMD 100 is used, the augmented reality caused by the firstimage, or the visualization of the operation of a part which cannot benormally viewed due to the second image can be used for training.Therefore, effective training can be performed.

Since the illusion feeling is strengthened in combination with thetactile stimulation, it is possible to perform the training moreeffectively.

The invention is not limited to the embodiments, examples, andmodification examples of the present specification, and can be realizedin various configurations without departing from the spirit thereof. Forexample, the technical features in the embodiments, examples, andmodification examples corresponding to the technical features in eachaspect described in the part of summary of the Invention can be replacedor combined as appropriate, in order to solve some or all of theabove-mentioned problems, or in order to achieve some or all of theaforementioned effects. Unless its technical features are described asessential here, the features can be deleted as appropriate. For example,the followings are exemplified.

FIG. 18 is a flowchart illustrating a first training process as amodification example. In the case of the modification example, unlikethe embodiment, S520 and S540 are not included. That is, it does notinclude a step for causing an illusion. Even though these steps are notincluded, it is possible to cause illusion by the first and secondimages. Therefore, when performing the first training processconveniently, S520 and S540 may be omitted as illustrated in FIG. 18.

The third arm which is an extra part may be connected to any part of thebody. For example, it may be connected to the left shoulder, belly,feet, or the like.

The extra part may be imaginary body parts other than the third arm. Forexample, it may be a tail, a wing, a third foot, a wheel, an endlesstrack, or the like. The wings may have a flapping structure like a wingof a bird, for example, or a structure like a helicopter rotating wing.

The extra part may be prosthesis (prosthetic hands or prosthetic feet)that supplement the missing limbs, prosthetic eyes functioning as asensory organ, or the like. This prosthetic eye may be configured tooperate to move the line of sight.

The method for determining the degree of activation of brain activitymay be changed. For example, the degree of activation of the electricalactivity of the brain may be used by using magneto encephalography(MEG), or it may be determined based on the amount of cerebral bloodflow. For measurement of cerebral blood flow, for example, near-infraredspectroscopy (NIRS) may be used, or functional magnetic resonanceimaging (fMRI) may be used.

During training using the HMD, the brain activity of the examinee maynot be measured. In this way, even in the case of using a measuringdevice (such as fMRI) that it is difficult to be used simultaneouslywith the HMD, the first and second training processes can be easilyperformed.

The effect measurement procedure (S700) and the specification procedure(S800) may be performed during the first training process. For example,even when the specific procedure is performed during the first trainingprocess, for example, the examinee which is to be subjected to thespecific procedure is a user who has undergone training to imagineoperating the extra part as in the embodiment.

The stimulation device 900 may apply thermal stimulation (temperaturestimulation) instead of pressure stimulation by vibration, contact, orthe like as tactile stimulation to the examinee. Alternatively, both thepressure stimulation and the temperature stimulation may be applied. Inorder to apply thermal stimulation, it is possible to adopt aconfiguration in which the stimulation device 900 includes a heatgenerating device such as a heater, or a cooling device such as aPeltier device or a water cooling device.

The stimulation applied to the examinee may be stimulation (that is,sound) to the auditory organ. For example, when an extra part operatesas the third arm, a wind noise may be heard by the examinee. Such asound may be realized by using an earphone provided in the HMD. Thestimulation to the auditory organ may be combined with the display ofthe illusion image or may be performed instead of the display of theillusion image.

Stimulation for illusion may be applied by an assistant. For example,when applying tactile stimulation, the assistant may touch theexaminee's back, at the timing when the extra part appears to touch theexaminee's back. The HMD may output to the external wirelesscommunication device, information indicating the timing at which theextra part appears to touch the examinee's back. An assistant may applytactile stimulation, with reference to its output.

The illusion image may be an image based on a first person viewpoint.For example, it may be an image that the extra part appears to touch theexaminee's arms.

The preliminary measurement procedure (S400) may not be performed. Inthis case, the effect measurement procedure may be performed based onwhether or not the activation area of the examinee to be determined isapproximately the same as the activation area in other examinees of apredetermined number of people or more.

An apparatus causing the examinee to view the display image in the firstand second training processes may be a non-transparent head mounteddisplay or a liquid crystal display. In the case of using these devices,the first image may be used.

In the case of using a non-transparent head mounted display, the firstimage may not be displayed, or the user's body may be displayed as thefirst image, in addition to the extra part.

In order to allow the examinee to view the display image in the firsttraining process while measuring the brain activity of the examinee byfMRI, an image may be transmitted from the outside using an opticalfiber, and displayed on a display device provided in the fMRI (a liquidcrystal display or the like).

In the second training process, the mode of displaying informationindicating the degree of activation of the corresponding area may bechanged. For example, the information may be displayed by the angle of arotating needle, or it may be displayed as a numerical value.

In the second training process, the user may be notified of theinformation indicating the degree of activation of the correspondingarea in a mode other than display by the HMD. For example, anotherdisplay device (a liquid crystal display or the like) may be used as anotifying unit, and information may be displayed on the display deviceso that the user can view it.

Alternatively, a sound output device outputting sound may be used as thenotifying unit. That is, the HMD or another device may output the soundas the information indicating the degree of activation of thecorresponding area. For example, as the degree of activation becomeshigher, the higher tone or the loud sound may be output.

In response to an instruction of the assistant, the operations describedas being performed by the examinee may voluntarily be performed by theexaminee.

The steps described as being performed by the assistant may be executedby the HMD or by a computer as an external device.

The entire disclosure of Japanese Patent Application No. 2016-031555filed Feb. 23, 2016 is expressly incorporated by reference herein.

What is claimed is:
 1. A training device comprising: a sensor configuredto: detect one of an electrical signal, a magnetic signal, anelectro-magnetic signal, or an optical signal indicating an amount of ablood flow or a cell ignition in brain areas of a brain of a user; andoutput a sensor result based on the detected signal; a memory configuredto store the sensor result and computer-readable instructions; a displayconfigured to display a notification image and a user activity image,the user activity image showing an extra part shown overlapping a firstactual body part of the user, and showing a second actual body part ofthe user; a stimulation device configured to generate a vibration, thestimulation device being configured to be attached to the second actualbody part of the user, the stimulation device being configured to applythe vibration to the second actual body part of the user; and aprocessor configured to execute the computer-readable instructions so asto: display, on the display, the user activity image from a viewpointthat shows the extra part moving with respect to the second actual bodypart of the user; determine a timing at which the extra part and thesecond actual body part of the user come into contact with each other inthe user activity image from the viewpoint; cause the stimulation deviceto generate the vibration to the second actual body part of the userwhen the processor determines the timing; determine, using the sensorresult, an area within the brain based on the amount of the blood flowor the cell ignition, the determined area indicating where a brainactivity occurs when the user activity image is displayed; and display(i) the notification image indicating a degree of the brain activity ofthe determined area in the brain based on the sensor result and (ii) theuser activity image on the display together so as to show a degree of anillusion of the user, the illusion of the extra part being a part of thebody of the user.
 2. The training device according to claim 1, whereinthe training device is a head mounted display, and the processor isconfigured to cause the display to display a virtual image for the user.3. The training device according to claim 1, wherein the processor isconfigured to cause the display to display a first image of the useractivity image and a second image of the user activity image, theprocessor is configured to create the first image from a viewpoint ofthe user, and the processor is configured to create the second imagefrom a viewpoint of a different person who observes the user.
 4. Thetraining device according to claim 3, wherein the training device is atransparent head mounted display, and wherein the first image is animage in which the extra part appears to be disposed relative to thebody of the user.
 5. The training device according to claim 1, whereinthe processor is configured to cause the display to display the useractivity image in which the extra part appears to apply the vibration,when the vibration is applied to at least one of a tactile organ and anauditory organ of the user as the second actual body part of the user.6. A training method for causing a processor to executecomputer-readable instructions stored in a memory, the method comprisingexecuting on the processor the steps of: detecting one of an electricalsignal, a magnetic signal, an electro-magnetic signal, or an opticalsignal indicating an amount of a blood flow or a cell ignition in brainareas of a brain of a user so as to output a sensor result based on thedetected signal; displaying, on a display, a user activity image showingan extra part shown as overlapping a first actual body part of the user,and showing a second actual body part of the user, the user activityimage being displayed from a viewpoint that shows the extra part movingwith respect to the second actual body part of the user; determining atiming at which the extra part and the second actual body part of theuser come into contact with each other in the user activity image fromthe viewpoint; causing a stimulation device to generate a vibration tothe second actual body part of the user when the processor determinesthe timing; determining, using the sensor result, an area within thebrain based on the amount of the blood flow or the cell ignition, thedetermined area indicating where a brain activity occurs when the useractivity image is displayed; and displaying (i) a notification imageindicating a degree of the brain activity of the determined area in thebrain based on the sensor result and (ii) the user activity image on thedisplay together so as to show a degree of an illusion of the user, theillusion of the extra part being a part of the body of the user.
 7. Acomputer program product embodying computer-readable instructions storedon a non-transitory computer-readable medium in which a program storedfor causing a computer to execute a process by a processor for atraining device so as to perform the steps of: detecting one of anelectrical signal, a magnetic signal, an electro-magnetic signal, or anoptical signal indicating an amount of a blood flow or a cell ignitionin brain areas of a brain of a user so as to output a sensor resultbased on the detected signal; displaying, on a display, a user activityimage showing an extra part shown as overlapping a first actual bodypart of the user, and showing a second actual body part of the user, theuser activity image being displayed from a viewpoint that shows theextra part moving with respect to the second actual body part of theuser; determining a timing at which the extra part and the second actualbody part of the user come into contact with each other in the useractivity image from the viewpoint; causing a stimulation device togenerate a vibration to the second actual body part of the user when theprocessor determines the timing; determining, using the sensor result,an area within the brain based on the amount of the blood flow or thecell ignition, the determined area indicating where a brain activityoccurs when the user activity image is displayed; and displaying (i) anotification image indicating a degree of the brain activity of thedetermined area in the brain based on the sensor result and (ii) theuser activity image on the display together so as to show a degree of anillusion of the user, the illusion of the extra part being a part of thebody of the user.